{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a62a7996",
   "metadata": {},
   "source": [
    "# DenseNet\n",
    "\n",
    "---\n",
    "\n",
    "### 背景和尝试解决的问题\n",
    "\n",
    "- **ResNet后的新挑战**：  \n",
    "  ResNet通过残差连接缓解了梯度消失问题，但每层仅与相邻层相加，特征复用效率有限。深层网络中，特征的逐层传递可能导致信息逐渐稀释。\n",
    "- **参数冗余问题**：  \n",
    "  传统CNN（如VGG、ResNet）通过堆叠层数提升性能，但参数利用率低，大量重复的特征提取导致计算成本高昂。\n",
    "- **梯度消失残留风险**：  \n",
    "  尽管ResNet改善了梯度传播，但在极深层网络中（如1000层以上），梯度衰减问题仍未完全消除。\n",
    "\n",
    "**DenseNet目标**：  \n",
    "通过密集跨层连接最大化特征复用，减少参数冗余，同时进一步优化梯度流动（由Gao Huang等人在2017年提出）。\n",
    "\n",
    "---\n",
    "\n",
    "![alt text](resources/densenet_arch.png \"Title\")\n",
    "\n",
    "## 创新点\n",
    "\n",
    "### 密集连接（Dense Connectivity）\n",
    "- **密集块（Dense Block）设计**：  \n",
    "  每一层的输入来自前面所有层的输出（例如第$L$层的输入为$[x_0, x_1, ..., x_{L-1}]$，其中$x_i$为第$i$层的特征图），通过**通道维度拼接**（Concatenation）而非ResNet的加法。\n",
    "- **特征复用与多样性**：  \n",
    "  每一层均可访问所有前置层的特征，避免重复提取，鼓励网络学习新特征（即“集体知识”）。\n",
    "\n",
    "### 关键组件设计\n",
    "- **增长率（Growth Rate $k$）**：  \n",
    "  控制每层输出特征图的通道数（例如$k=32$），密集块内每层的输出通道固定为$k$，但拼接后的输入通道数线性增长。\n",
    "- **过渡层（Transition Layer）**：  \n",
    "  位于密集块之间，包含：\n",
    "  - 1×1卷积（压缩通道数）\n",
    "  - 2×2平均池化（降采样）\n",
    "- **瓶颈层（Bottleneck Layer）**：  \n",
    "  在密集块内，每层先通过1×1卷积降维（减少计算量），再进行3×3卷积。\n",
    "\n",
    "### 优势\n",
    "- **参数高效**：  \n",
    "  DenseNet-201（20M参数）在ImageNet上性能优于ResNet-152（60M参数）。\n",
    "- **梯度流动增强**：  \n",
    "  反向传播时梯度可直达所有前置层，彻底消除梯度消失问题。\n",
    "- **隐式深度监督**：  \n",
    "  浅层特征直接参与深层计算，相当于自动引入中间监督信号。\n",
    "\n",
    "![alt text](resources/densenet_loss_surface.png \"Title\")\n",
    "\n",
    "[Visualizing the Loss Landscape of Neural Nets](https://arxiv.org/pdf/1712.09913)也指出,DenseNet的损失函数更加平滑,更容易收敛。\n",
    "\n",
    "![alt text](resources/densenet_detail.png \"Title\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63089980",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ae655c5f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Use device:  cuda\n"
     ]
    }
   ],
   "source": [
    "# 自动重新加载外部module，使得修改代码之后无需重新import\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "from hdd.device.utils import get_device\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torchvision import datasets, transforms\n",
    "\n",
    "# 设置训练数据的路径\n",
    "DATA_ROOT = \"~/workspace/hands-dirty-on-dl/dataset\"\n",
    "# 设置TensorBoard的路径\n",
    "TENSORBOARD_ROOT = \"~/workspace/hands-dirty-on-dl/dataset\"\n",
    "# 设置预训练模型参数路径\n",
    "TORCH_HUB_PATH = \"~/workspace/hands-dirty-on-dl/pretrained_models\"\n",
    "torch.hub.set_dir(TORCH_HUB_PATH)\n",
    "# 挑选最合适的训练设备\n",
    "DEVICE = get_device([\"cuda\", \"cpu\"])\n",
    "print(\"Use device: \", DEVICE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e02b05f",
   "metadata": {},
   "source": [
    "## Experiment on cifar10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e6f55268",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n",
      "Basic Info of train dataset: \n",
      " Dataset CIFAR10\n",
      "    Number of datapoints: 50000\n",
      "    Root location: /home/tf/workspace/hands-dirty-on-dl/dataset\n",
      "    Split: Train\n",
      "    StandardTransform\n",
      "Transform: Compose(\n",
      "               Pad(padding=4, fill=0, padding_mode=constant)\n",
      "               RandomRotation(degrees=[-3.0, 3.0], interpolation=nearest, expand=False, fill=0)\n",
      "               RandomCrop(size=(32, 32), padding=None)\n",
      "               RandomHorizontalFlip(p=0.5)\n",
      "               ToTensor()\n",
      "               Normalize(mean=[0.50707516, 0.48654887, 0.44091784], std=[0.26733429, 0.25643846, 0.27615047])\n",
      "           )\n",
      "Basic Info of test dataset: \n",
      " Dataset CIFAR10\n",
      "    Number of datapoints: 10000\n",
      "    Root location: /home/tf/workspace/hands-dirty-on-dl/dataset\n",
      "    Split: Test\n",
      "    StandardTransform\n",
      "Transform: Compose(\n",
      "               ToTensor()\n",
      "               Normalize(mean=[0.50707516, 0.48654887, 0.44091784], std=[0.26733429, 0.25643846, 0.27615047])\n",
      "           )\n"
     ]
    }
   ],
   "source": [
    "# 我们提前计算好了训练数据集上的均值和方差\n",
    "TRAIN_MEAN = [0.50707516, 0.48654887, 0.44091784]\n",
    "TRAIN_STD = [0.26733429, 0.25643846, 0.27615047]\n",
    "\n",
    "train_dataset_transforms = transforms.Compose(\n",
    "    [\n",
    "        transforms.Pad(4),\n",
    "        transforms.RandomRotation(3),\n",
    "        transforms.RandomCrop(32),\n",
    "        transforms.RandomHorizontalFlip(),\n",
    "        transforms.ToTensor(),\n",
    "        transforms.Normalize(mean=TRAIN_MEAN, std=TRAIN_STD),\n",
    "    ]\n",
    ")\n",
    "# 加载数据集\n",
    "train_dataset = datasets.CIFAR10(\n",
    "    root=DATA_ROOT,\n",
    "    train=True,\n",
    "    transform=train_dataset_transforms,\n",
    "    download=True,\n",
    ")\n",
    "val_dataset = datasets.CIFAR10(\n",
    "    root=DATA_ROOT,\n",
    "    train=False,\n",
    "    transform=transforms.Compose(\n",
    "        [transforms.ToTensor(), transforms.Normalize(TRAIN_MEAN, TRAIN_STD)]\n",
    "    ),\n",
    "    download=True,\n",
    ")\n",
    "print(\"Basic Info of train dataset: \\n\", train_dataset)\n",
    "print(\"Basic Info of test dataset: \\n\", val_dataset)\n",
    "BATCH_SIZE = 64\n",
    "train_dataloader = torch.utils.data.DataLoader(\n",
    "    train_dataset,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    shuffle=True,\n",
    "    num_workers=4,\n",
    ")\n",
    "val_dataloader = torch.utils.data.DataLoader(\n",
    "    val_dataset,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    shuffle=False,\n",
    "    num_workers=4,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ed4d96ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "from hdd.train.classification_utils import (\n",
    "    naive_train_classification_model,\n",
    "    eval_image_classifier,\n",
    ")\n",
    "from hdd.models.nn_utils import count_trainable_parameter\n",
    "\n",
    "\n",
    "def train_net(\n",
    "    net,\n",
    "    train_dataloader,\n",
    "    val_dataloader,\n",
    "    lr,\n",
    "    weight_decay,\n",
    "    step_size=30,\n",
    "    gamma=0.1,\n",
    "    max_epochs=130,\n",
    ") -> dict[str, list[float]]:\n",
    "    criteria = nn.CrossEntropyLoss()\n",
    "    optimizer = torch.optim.SGD(\n",
    "        net.parameters(), lr=lr, momentum=0.9, weight_decay=weight_decay\n",
    "    )\n",
    "    scheduler = torch.optim.lr_scheduler.StepLR(\n",
    "        optimizer, step_size=step_size, gamma=gamma, last_epoch=-1\n",
    "    )\n",
    "    training_stats = naive_train_classification_model(\n",
    "        net,\n",
    "        criteria,\n",
    "        max_epochs,\n",
    "        train_dataloader,\n",
    "        val_dataloader,\n",
    "        DEVICE,\n",
    "        optimizer,\n",
    "        scheduler,\n",
    "        verbose=True,\n",
    "    )\n",
    "    return training_stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f6c774c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/130 Train Loss: 1.6175 Accuracy: 0.4088 Time: 17.85499  | Val Loss: 1.5728 Accuracy: 0.4665\n",
      "Epoch: 2/130 Train Loss: 1.1472 Accuracy: 0.5892 Time: 17.97379  | Val Loss: 1.0788 Accuracy: 0.6284\n",
      "Epoch: 3/130 Train Loss: 0.9404 Accuracy: 0.6672 Time: 17.32507  | Val Loss: 1.0798 Accuracy: 0.6537\n",
      "Epoch: 4/130 Train Loss: 0.8059 Accuracy: 0.7165 Time: 17.59543  | Val Loss: 1.0048 Accuracy: 0.6859\n",
      "Epoch: 5/130 Train Loss: 0.7054 Accuracy: 0.7538 Time: 17.18794  | Val Loss: 0.7277 Accuracy: 0.7547\n",
      "Epoch: 6/130 Train Loss: 0.6366 Accuracy: 0.7780 Time: 18.07373  | Val Loss: 0.6327 Accuracy: 0.7863\n",
      "Epoch: 7/130 Train Loss: 0.5904 Accuracy: 0.7950 Time: 17.17058  | Val Loss: 0.6532 Accuracy: 0.7801\n",
      "Epoch: 8/130 Train Loss: 0.5569 Accuracy: 0.8077 Time: 17.72657  | Val Loss: 0.6275 Accuracy: 0.7944\n",
      "Epoch: 9/130 Train Loss: 0.5217 Accuracy: 0.8207 Time: 17.86374  | Val Loss: 0.5705 Accuracy: 0.8123\n",
      "Epoch: 10/130 Train Loss: 0.5014 Accuracy: 0.8274 Time: 16.90706  | Val Loss: 0.5395 Accuracy: 0.8223\n",
      "Epoch: 11/130 Train Loss: 0.4839 Accuracy: 0.8335 Time: 17.25053  | Val Loss: 0.5789 Accuracy: 0.8156\n",
      "Epoch: 12/130 Train Loss: 0.4640 Accuracy: 0.8396 Time: 16.64584  | Val Loss: 0.6089 Accuracy: 0.8124\n",
      "Epoch: 13/130 Train Loss: 0.4459 Accuracy: 0.8467 Time: 15.97254  | Val Loss: 0.5330 Accuracy: 0.8340\n",
      "Epoch: 14/130 Train Loss: 0.4329 Accuracy: 0.8514 Time: 17.06487  | Val Loss: 0.5097 Accuracy: 0.8430\n",
      "Epoch: 15/130 Train Loss: 0.4266 Accuracy: 0.8526 Time: 17.04282  | Val Loss: 0.5545 Accuracy: 0.8190\n",
      "Epoch: 16/130 Train Loss: 0.4169 Accuracy: 0.8562 Time: 16.02553  | Val Loss: 0.4363 Accuracy: 0.8550\n",
      "Epoch: 17/130 Train Loss: 0.4062 Accuracy: 0.8600 Time: 16.88606  | Val Loss: 0.4618 Accuracy: 0.8551\n",
      "Epoch: 18/130 Train Loss: 0.3991 Accuracy: 0.8603 Time: 18.71482  | Val Loss: 0.5062 Accuracy: 0.8375\n",
      "Epoch: 19/130 Train Loss: 0.3925 Accuracy: 0.8622 Time: 17.07730  | Val Loss: 0.3931 Accuracy: 0.8732\n",
      "Epoch: 20/130 Train Loss: 0.3846 Accuracy: 0.8667 Time: 19.48138  | Val Loss: 0.4526 Accuracy: 0.8474\n",
      "Epoch: 21/130 Train Loss: 0.3743 Accuracy: 0.8708 Time: 18.05641  | Val Loss: 0.4797 Accuracy: 0.8491\n",
      "Epoch: 22/130 Train Loss: 0.3737 Accuracy: 0.8707 Time: 17.77854  | Val Loss: 0.4779 Accuracy: 0.8384\n",
      "Epoch: 23/130 Train Loss: 0.3671 Accuracy: 0.8730 Time: 17.59935  | Val Loss: 0.5122 Accuracy: 0.8413\n",
      "Epoch: 24/130 Train Loss: 0.3634 Accuracy: 0.8752 Time: 16.17022  | Val Loss: 0.5106 Accuracy: 0.8320\n",
      "Epoch: 25/130 Train Loss: 0.3659 Accuracy: 0.8732 Time: 17.20029  | Val Loss: 0.5752 Accuracy: 0.8131\n",
      "Epoch: 26/130 Train Loss: 0.3554 Accuracy: 0.8761 Time: 17.75863  | Val Loss: 0.4190 Accuracy: 0.8603\n",
      "Epoch: 27/130 Train Loss: 0.3470 Accuracy: 0.8812 Time: 16.49590  | Val Loss: 0.4791 Accuracy: 0.8431\n",
      "Epoch: 28/130 Train Loss: 0.3492 Accuracy: 0.8782 Time: 17.31503  | Val Loss: 0.4420 Accuracy: 0.8545\n",
      "Epoch: 29/130 Train Loss: 0.3402 Accuracy: 0.8823 Time: 15.82098  | Val Loss: 0.3617 Accuracy: 0.8800\n",
      "Epoch: 30/130 Train Loss: 0.3439 Accuracy: 0.8823 Time: 16.76108  | Val Loss: 0.6043 Accuracy: 0.8239\n",
      "Epoch: 31/130 Train Loss: 0.2212 Accuracy: 0.9253 Time: 17.64738  | Val Loss: 0.2420 Accuracy: 0.9166\n",
      "Epoch: 32/130 Train Loss: 0.1907 Accuracy: 0.9350 Time: 17.49658  | Val Loss: 0.2299 Accuracy: 0.9214\n",
      "Epoch: 33/130 Train Loss: 0.1703 Accuracy: 0.9418 Time: 17.33970  | Val Loss: 0.2250 Accuracy: 0.9242\n",
      "Epoch: 34/130 Train Loss: 0.1647 Accuracy: 0.9440 Time: 16.24879  | Val Loss: 0.2252 Accuracy: 0.9239\n",
      "Epoch: 35/130 Train Loss: 0.1580 Accuracy: 0.9460 Time: 16.32794  | Val Loss: 0.2311 Accuracy: 0.9262\n",
      "Epoch: 36/130 Train Loss: 0.1522 Accuracy: 0.9476 Time: 17.46321  | Val Loss: 0.2317 Accuracy: 0.9250\n",
      "Epoch: 37/130 Train Loss: 0.1475 Accuracy: 0.9490 Time: 17.45555  | Val Loss: 0.2282 Accuracy: 0.9258\n",
      "Epoch: 38/130 Train Loss: 0.1419 Accuracy: 0.9507 Time: 17.61327  | Val Loss: 0.2334 Accuracy: 0.9263\n",
      "Epoch: 39/130 Train Loss: 0.1368 Accuracy: 0.9525 Time: 17.39770  | Val Loss: 0.2419 Accuracy: 0.9247\n",
      "Epoch: 40/130 Train Loss: 0.1337 Accuracy: 0.9534 Time: 17.30510  | Val Loss: 0.2324 Accuracy: 0.9266\n",
      "Epoch: 41/130 Train Loss: 0.1295 Accuracy: 0.9542 Time: 17.37037  | Val Loss: 0.2343 Accuracy: 0.9277\n",
      "Epoch: 42/130 Train Loss: 0.1293 Accuracy: 0.9546 Time: 18.31499  | Val Loss: 0.2320 Accuracy: 0.9284\n",
      "Epoch: 43/130 Train Loss: 0.1212 Accuracy: 0.9584 Time: 17.15615  | Val Loss: 0.2406 Accuracy: 0.9268\n",
      "Epoch: 44/130 Train Loss: 0.1204 Accuracy: 0.9576 Time: 17.33663  | Val Loss: 0.2410 Accuracy: 0.9286\n",
      "Epoch: 45/130 Train Loss: 0.1192 Accuracy: 0.9587 Time: 17.58929  | Val Loss: 0.2381 Accuracy: 0.9281\n",
      "Epoch: 46/130 Train Loss: 0.1174 Accuracy: 0.9585 Time: 15.81029  | Val Loss: 0.2388 Accuracy: 0.9268\n",
      "Epoch: 47/130 Train Loss: 0.1136 Accuracy: 0.9599 Time: 15.83944  | Val Loss: 0.2451 Accuracy: 0.9267\n",
      "Epoch: 48/130 Train Loss: 0.1090 Accuracy: 0.9625 Time: 16.40035  | Val Loss: 0.2406 Accuracy: 0.9287\n",
      "Epoch: 49/130 Train Loss: 0.1062 Accuracy: 0.9626 Time: 16.80176  | Val Loss: 0.2735 Accuracy: 0.9190\n",
      "Epoch: 50/130 Train Loss: 0.1072 Accuracy: 0.9619 Time: 16.75996  | Val Loss: 0.2600 Accuracy: 0.9240\n",
      "Epoch: 51/130 Train Loss: 0.1049 Accuracy: 0.9632 Time: 15.92286  | Val Loss: 0.2560 Accuracy: 0.9233\n",
      "Epoch: 52/130 Train Loss: 0.1039 Accuracy: 0.9644 Time: 16.09149  | Val Loss: 0.2563 Accuracy: 0.9253\n",
      "Epoch: 53/130 Train Loss: 0.0994 Accuracy: 0.9659 Time: 17.06934  | Val Loss: 0.2739 Accuracy: 0.9230\n",
      "Epoch: 54/130 Train Loss: 0.0991 Accuracy: 0.9653 Time: 16.48189  | Val Loss: 0.2573 Accuracy: 0.9265\n",
      "Epoch: 55/130 Train Loss: 0.0986 Accuracy: 0.9663 Time: 16.14640  | Val Loss: 0.2627 Accuracy: 0.9260\n",
      "Epoch: 56/130 Train Loss: 0.0993 Accuracy: 0.9648 Time: 15.76829  | Val Loss: 0.2810 Accuracy: 0.9206\n",
      "Epoch: 57/130 Train Loss: 0.0956 Accuracy: 0.9658 Time: 16.56574  | Val Loss: 0.2723 Accuracy: 0.9266\n",
      "Epoch: 58/130 Train Loss: 0.0960 Accuracy: 0.9665 Time: 16.59709  | Val Loss: 0.2758 Accuracy: 0.9229\n",
      "Epoch: 59/130 Train Loss: 0.0945 Accuracy: 0.9668 Time: 15.95492  | Val Loss: 0.2580 Accuracy: 0.9257\n",
      "Epoch: 60/130 Train Loss: 0.0946 Accuracy: 0.9661 Time: 15.82054  | Val Loss: 0.2798 Accuracy: 0.9222\n",
      "Epoch: 61/130 Train Loss: 0.0745 Accuracy: 0.9751 Time: 15.84522  | Val Loss: 0.2560 Accuracy: 0.9278\n",
      "Epoch: 62/130 Train Loss: 0.0682 Accuracy: 0.9781 Time: 16.04370  | Val Loss: 0.2456 Accuracy: 0.9306\n",
      "Epoch: 63/130 Train Loss: 0.0649 Accuracy: 0.9779 Time: 15.06657  | Val Loss: 0.2488 Accuracy: 0.9300\n",
      "Epoch: 64/130 Train Loss: 0.0636 Accuracy: 0.9786 Time: 15.09408  | Val Loss: 0.2532 Accuracy: 0.9292\n",
      "Epoch: 65/130 Train Loss: 0.0635 Accuracy: 0.9786 Time: 15.75423  | Val Loss: 0.2520 Accuracy: 0.9293\n",
      "Epoch: 66/130 Train Loss: 0.0603 Accuracy: 0.9803 Time: 16.12811  | Val Loss: 0.2543 Accuracy: 0.9305\n",
      "Epoch: 67/130 Train Loss: 0.0590 Accuracy: 0.9800 Time: 16.23734  | Val Loss: 0.2531 Accuracy: 0.9305\n",
      "Epoch: 68/130 Train Loss: 0.0584 Accuracy: 0.9802 Time: 18.13222  | Val Loss: 0.2527 Accuracy: 0.9310\n",
      "Epoch: 69/130 Train Loss: 0.0570 Accuracy: 0.9816 Time: 17.13042  | Val Loss: 0.2475 Accuracy: 0.9311\n",
      "Epoch: 70/130 Train Loss: 0.0579 Accuracy: 0.9804 Time: 17.89006  | Val Loss: 0.2542 Accuracy: 0.9308\n",
      "Epoch: 71/130 Train Loss: 0.0561 Accuracy: 0.9812 Time: 17.37882  | Val Loss: 0.2505 Accuracy: 0.9312\n",
      "Epoch: 72/130 Train Loss: 0.0560 Accuracy: 0.9812 Time: 18.28207  | Val Loss: 0.2549 Accuracy: 0.9307\n",
      "Epoch: 73/130 Train Loss: 0.0545 Accuracy: 0.9817 Time: 17.79963  | Val Loss: 0.2542 Accuracy: 0.9304\n",
      "Epoch: 74/130 Train Loss: 0.0550 Accuracy: 0.9816 Time: 16.33369  | Val Loss: 0.2582 Accuracy: 0.9295\n",
      "Epoch: 75/130 Train Loss: 0.0542 Accuracy: 0.9818 Time: 16.77268  | Val Loss: 0.2497 Accuracy: 0.9311\n",
      "Epoch: 76/130 Train Loss: 0.0531 Accuracy: 0.9828 Time: 17.52145  | Val Loss: 0.2523 Accuracy: 0.9308\n",
      "Epoch: 77/130 Train Loss: 0.0527 Accuracy: 0.9832 Time: 18.07183  | Val Loss: 0.2568 Accuracy: 0.9297\n",
      "Epoch: 78/130 Train Loss: 0.0547 Accuracy: 0.9811 Time: 17.06334  | Val Loss: 0.2599 Accuracy: 0.9301\n",
      "Epoch: 79/130 Train Loss: 0.0546 Accuracy: 0.9820 Time: 17.04706  | Val Loss: 0.2600 Accuracy: 0.9303\n",
      "Epoch: 80/130 Train Loss: 0.0519 Accuracy: 0.9828 Time: 16.27404  | Val Loss: 0.2631 Accuracy: 0.9314\n",
      "Epoch: 81/130 Train Loss: 0.0519 Accuracy: 0.9829 Time: 17.16863  | Val Loss: 0.2593 Accuracy: 0.9307\n",
      "Epoch: 82/130 Train Loss: 0.0520 Accuracy: 0.9834 Time: 16.94590  | Val Loss: 0.2579 Accuracy: 0.9314\n",
      "Epoch: 83/130 Train Loss: 0.0493 Accuracy: 0.9840 Time: 17.33310  | Val Loss: 0.2595 Accuracy: 0.9302\n",
      "Epoch: 84/130 Train Loss: 0.0499 Accuracy: 0.9832 Time: 16.56692  | Val Loss: 0.2651 Accuracy: 0.9309\n",
      "Epoch: 85/130 Train Loss: 0.0511 Accuracy: 0.9830 Time: 17.58776  | Val Loss: 0.2602 Accuracy: 0.9308\n",
      "Epoch: 86/130 Train Loss: 0.0497 Accuracy: 0.9840 Time: 16.01082  | Val Loss: 0.2606 Accuracy: 0.9307\n",
      "Epoch: 87/130 Train Loss: 0.0489 Accuracy: 0.9830 Time: 14.95865  | Val Loss: 0.2578 Accuracy: 0.9310\n",
      "Epoch: 88/130 Train Loss: 0.0488 Accuracy: 0.9833 Time: 15.47596  | Val Loss: 0.2655 Accuracy: 0.9306\n",
      "Epoch: 89/130 Train Loss: 0.0498 Accuracy: 0.9834 Time: 15.49267  | Val Loss: 0.2637 Accuracy: 0.9295\n",
      "Epoch: 90/130 Train Loss: 0.0475 Accuracy: 0.9848 Time: 15.79668  | Val Loss: 0.2648 Accuracy: 0.9297\n",
      "Epoch: 91/130 Train Loss: 0.0471 Accuracy: 0.9851 Time: 15.60142  | Val Loss: 0.2601 Accuracy: 0.9304\n",
      "Epoch: 92/130 Train Loss: 0.0465 Accuracy: 0.9846 Time: 15.97014  | Val Loss: 0.2621 Accuracy: 0.9297\n",
      "Epoch: 93/130 Train Loss: 0.0458 Accuracy: 0.9849 Time: 15.83169  | Val Loss: 0.2681 Accuracy: 0.9294\n",
      "Epoch: 94/130 Train Loss: 0.0464 Accuracy: 0.9854 Time: 15.87663  | Val Loss: 0.2668 Accuracy: 0.9297\n",
      "Epoch: 95/130 Train Loss: 0.0448 Accuracy: 0.9859 Time: 15.03565  | Val Loss: 0.2650 Accuracy: 0.9305\n",
      "Epoch: 96/130 Train Loss: 0.0433 Accuracy: 0.9863 Time: 15.57798  | Val Loss: 0.2647 Accuracy: 0.9302\n",
      "Epoch: 97/130 Train Loss: 0.0443 Accuracy: 0.9854 Time: 15.69872  | Val Loss: 0.2617 Accuracy: 0.9307\n",
      "Epoch: 98/130 Train Loss: 0.0460 Accuracy: 0.9847 Time: 15.64078  | Val Loss: 0.2629 Accuracy: 0.9310\n",
      "Epoch: 99/130 Train Loss: 0.0457 Accuracy: 0.9852 Time: 16.70146  | Val Loss: 0.2674 Accuracy: 0.9311\n",
      "Epoch: 100/130 Train Loss: 0.0450 Accuracy: 0.9855 Time: 15.42222  | Val Loss: 0.2634 Accuracy: 0.9314\n",
      "Epoch: 101/130 Train Loss: 0.0433 Accuracy: 0.9866 Time: 15.05576  | Val Loss: 0.2660 Accuracy: 0.9313\n",
      "Epoch: 102/130 Train Loss: 0.0446 Accuracy: 0.9855 Time: 16.14492  | Val Loss: 0.2654 Accuracy: 0.9307\n",
      "Epoch: 103/130 Train Loss: 0.0457 Accuracy: 0.9852 Time: 16.15665  | Val Loss: 0.2599 Accuracy: 0.9301\n",
      "Epoch: 104/130 Train Loss: 0.0465 Accuracy: 0.9847 Time: 15.90504  | Val Loss: 0.2622 Accuracy: 0.9303\n",
      "Epoch: 105/130 Train Loss: 0.0456 Accuracy: 0.9848 Time: 15.47218  | Val Loss: 0.2651 Accuracy: 0.9303\n",
      "Epoch: 106/130 Train Loss: 0.0441 Accuracy: 0.9859 Time: 15.56456  | Val Loss: 0.2660 Accuracy: 0.9303\n",
      "Epoch: 107/130 Train Loss: 0.0453 Accuracy: 0.9854 Time: 15.80914  | Val Loss: 0.2657 Accuracy: 0.9310\n",
      "Epoch: 108/130 Train Loss: 0.0439 Accuracy: 0.9865 Time: 15.26936  | Val Loss: 0.2670 Accuracy: 0.9307\n",
      "Epoch: 109/130 Train Loss: 0.0449 Accuracy: 0.9854 Time: 15.24775  | Val Loss: 0.2654 Accuracy: 0.9305\n",
      "Epoch: 110/130 Train Loss: 0.0458 Accuracy: 0.9854 Time: 15.42245  | Val Loss: 0.2689 Accuracy: 0.9293\n",
      "Epoch: 111/130 Train Loss: 0.0450 Accuracy: 0.9855 Time: 15.44902  | Val Loss: 0.2624 Accuracy: 0.9310\n",
      "Epoch: 112/130 Train Loss: 0.0444 Accuracy: 0.9851 Time: 15.51742  | Val Loss: 0.2654 Accuracy: 0.9304\n",
      "Epoch: 113/130 Train Loss: 0.0471 Accuracy: 0.9848 Time: 16.10350  | Val Loss: 0.2684 Accuracy: 0.9312\n",
      "Epoch: 114/130 Train Loss: 0.0435 Accuracy: 0.9868 Time: 16.51623  | Val Loss: 0.2652 Accuracy: 0.9304\n",
      "Epoch: 115/130 Train Loss: 0.0449 Accuracy: 0.9854 Time: 15.52615  | Val Loss: 0.2626 Accuracy: 0.9306\n",
      "Epoch: 116/130 Train Loss: 0.0454 Accuracy: 0.9852 Time: 15.38119  | Val Loss: 0.2662 Accuracy: 0.9299\n",
      "Epoch: 117/130 Train Loss: 0.0446 Accuracy: 0.9860 Time: 15.43807  | Val Loss: 0.2665 Accuracy: 0.9307\n",
      "Epoch: 118/130 Train Loss: 0.0456 Accuracy: 0.9851 Time: 15.13996  | Val Loss: 0.2650 Accuracy: 0.9298\n",
      "Epoch: 119/130 Train Loss: 0.0463 Accuracy: 0.9848 Time: 15.40098  | Val Loss: 0.2651 Accuracy: 0.9297\n",
      "Epoch: 120/130 Train Loss: 0.0439 Accuracy: 0.9856 Time: 15.47718  | Val Loss: 0.2632 Accuracy: 0.9307\n",
      "Epoch: 121/130 Train Loss: 0.0435 Accuracy: 0.9860 Time: 16.85463  | Val Loss: 0.2660 Accuracy: 0.9312\n",
      "Epoch: 122/130 Train Loss: 0.0453 Accuracy: 0.9852 Time: 18.47062  | Val Loss: 0.2705 Accuracy: 0.9299\n",
      "Epoch: 123/130 Train Loss: 0.0453 Accuracy: 0.9850 Time: 16.71521  | Val Loss: 0.2696 Accuracy: 0.9312\n",
      "Epoch: 124/130 Train Loss: 0.0449 Accuracy: 0.9852 Time: 16.79038  | Val Loss: 0.2663 Accuracy: 0.9311\n",
      "Epoch: 125/130 Train Loss: 0.0435 Accuracy: 0.9852 Time: 16.46280  | Val Loss: 0.2679 Accuracy: 0.9303\n",
      "Epoch: 126/130 Train Loss: 0.0449 Accuracy: 0.9852 Time: 17.40340  | Val Loss: 0.2659 Accuracy: 0.9307\n",
      "Epoch: 127/130 Train Loss: 0.0449 Accuracy: 0.9853 Time: 17.68334  | Val Loss: 0.2613 Accuracy: 0.9309\n",
      "Epoch: 128/130 Train Loss: 0.0429 Accuracy: 0.9862 Time: 18.21422  | Val Loss: 0.2651 Accuracy: 0.9307\n",
      "Epoch: 129/130 Train Loss: 0.0454 Accuracy: 0.9851 Time: 17.20192  | Val Loss: 0.2666 Accuracy: 0.9303\n",
      "Epoch: 130/130 Train Loss: 0.0450 Accuracy: 0.9858 Time: 18.51317  | Val Loss: 0.2669 Accuracy: 0.9304\n",
      "#Parameter: 1261018 Accuracy: 0.9304\n"
     ]
    }
   ],
   "source": [
    "from hdd.models.cnn.densenet import (\n",
    "    DenseNetSmall40,\n",
    ")\n",
    "\n",
    "net = DenseNetSmall40(num_classes=10, dropout=0.2, growth_rate=12).to(DEVICE)\n",
    "dense40_stats = train_net(\n",
    "    net,\n",
    "    train_dataloader,\n",
    "    val_dataloader,\n",
    "    lr=0.1,\n",
    "    weight_decay=1e-4,\n",
    ")\n",
    "\n",
    "eval_result = eval_image_classifier(net, val_dataloader.dataset, DEVICE)\n",
    "ss = [result.gt_label == result.predicted_label for result in eval_result]\n",
    "print(f\"#Parameter: {count_trainable_parameter(net)} Accuracy: {sum(ss) / len(ss)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee8b9e10",
   "metadata": {},
   "source": [
    "## Experiment on imagenette"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "654e4876",
   "metadata": {},
   "outputs": [],
   "source": [
    "from hdd.dataset.imagenette_in_memory import ImagenetteInMemory\n",
    "from hdd.data_util.transforms import RandomResize\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "TRAIN_MEAN = [0.4625, 0.4580, 0.4295]\n",
    "TRAIN_STD = [0.2452, 0.2390, 0.2469]\n",
    "train_dataset_transforms = transforms.Compose(\n",
    "    [\n",
    "        RandomResize([256, 296, 384]),  # 随机在三个size中选择一个进行resize\n",
    "        transforms.RandomRotation(10),\n",
    "        transforms.RandomCrop(224),\n",
    "        transforms.RandomHorizontalFlip(),\n",
    "        transforms.ToTensor(),\n",
    "        transforms.Normalize(mean=TRAIN_MEAN, std=TRAIN_STD),\n",
    "    ]\n",
    ")\n",
    "val_dataset_transforms = transforms.Compose(\n",
    "    [\n",
    "        transforms.Resize(256),\n",
    "        transforms.CenterCrop(224),\n",
    "        transforms.ToTensor(),\n",
    "        transforms.Normalize(mean=TRAIN_MEAN, std=TRAIN_STD),\n",
    "    ]\n",
    ")\n",
    "train_dataset = ImagenetteInMemory(\n",
    "    root=DATA_ROOT,\n",
    "    split=\"train\",\n",
    "    size=\"full\",\n",
    "    download=True,\n",
    "    transform=train_dataset_transforms,\n",
    ")\n",
    "val_dataset = ImagenetteInMemory(\n",
    "    root=DATA_ROOT,\n",
    "    split=\"val\",\n",
    "    size=\"full\",\n",
    "    download=True,\n",
    "    transform=val_dataset_transforms,\n",
    ")\n",
    "\n",
    "\n",
    "def build_dataloader(batch_size, train_dataset, val_dataset):\n",
    "    train_dataloader = DataLoader(\n",
    "        train_dataset, batch_size=batch_size, shuffle=True, num_workers=8\n",
    "    )\n",
    "    val_dataloader = DataLoader(\n",
    "        val_dataset, batch_size=batch_size, shuffle=False, num_workers=8\n",
    "    )\n",
    "    return train_dataloader, val_dataloader\n",
    "\n",
    "\n",
    "train_dataloader, val_dataloader = build_dataloader(32, train_dataset, val_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "edc71657",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/130 Train Loss: 2.0198 Accuracy: 0.3108 Time: 8.66043  | Val Loss: 1.7831 Accuracy: 0.4433\n",
      "Epoch: 2/130 Train Loss: 1.5586 Accuracy: 0.4804 Time: 8.67856  | Val Loss: 1.4234 Accuracy: 0.5335\n",
      "Epoch: 3/130 Train Loss: 1.3372 Accuracy: 0.5606 Time: 8.72212  | Val Loss: 1.1452 Accuracy: 0.6285\n",
      "Epoch: 4/130 Train Loss: 1.2100 Accuracy: 0.6051 Time: 8.67810  | Val Loss: 1.0056 Accuracy: 0.6818\n",
      "Epoch: 5/130 Train Loss: 1.1018 Accuracy: 0.6403 Time: 8.78715  | Val Loss: 0.9796 Accuracy: 0.6838\n",
      "Epoch: 6/130 Train Loss: 1.0276 Accuracy: 0.6647 Time: 8.71148  | Val Loss: 0.8829 Accuracy: 0.7090\n",
      "Epoch: 7/130 Train Loss: 0.9668 Accuracy: 0.6833 Time: 8.70774  | Val Loss: 0.8575 Accuracy: 0.7236\n",
      "Epoch: 8/130 Train Loss: 0.9285 Accuracy: 0.6986 Time: 8.98798  | Val Loss: 0.7858 Accuracy: 0.7496\n",
      "Epoch: 9/130 Train Loss: 0.8761 Accuracy: 0.7154 Time: 8.78934  | Val Loss: 0.8444 Accuracy: 0.7210\n",
      "Epoch: 10/130 Train Loss: 0.8507 Accuracy: 0.7274 Time: 8.80131  | Val Loss: 0.7464 Accuracy: 0.7636\n",
      "Epoch: 11/130 Train Loss: 0.8059 Accuracy: 0.7366 Time: 8.83672  | Val Loss: 0.8262 Accuracy: 0.7383\n",
      "Epoch: 12/130 Train Loss: 0.7705 Accuracy: 0.7519 Time: 8.83671  | Val Loss: 0.7990 Accuracy: 0.7536\n",
      "Epoch: 13/130 Train Loss: 0.7516 Accuracy: 0.7574 Time: 9.01411  | Val Loss: 0.7131 Accuracy: 0.7745\n",
      "Epoch: 14/130 Train Loss: 0.7389 Accuracy: 0.7625 Time: 8.93139  | Val Loss: 0.7032 Accuracy: 0.7898\n",
      "Epoch: 15/130 Train Loss: 0.7049 Accuracy: 0.7741 Time: 8.87039  | Val Loss: 0.6880 Accuracy: 0.7860\n",
      "Epoch: 16/130 Train Loss: 0.6841 Accuracy: 0.7814 Time: 8.89906  | Val Loss: 0.5444 Accuracy: 0.8265\n",
      "Epoch: 17/130 Train Loss: 0.6752 Accuracy: 0.7795 Time: 8.78768  | Val Loss: 0.5751 Accuracy: 0.8155\n",
      "Epoch: 18/130 Train Loss: 0.6587 Accuracy: 0.7869 Time: 8.90469  | Val Loss: 0.8256 Accuracy: 0.7462\n",
      "Epoch: 19/130 Train Loss: 0.6531 Accuracy: 0.7914 Time: 8.70379  | Val Loss: 0.5923 Accuracy: 0.8117\n",
      "Epoch: 20/130 Train Loss: 0.6406 Accuracy: 0.7915 Time: 8.86467  | Val Loss: 0.5834 Accuracy: 0.8125\n",
      "Epoch: 21/130 Train Loss: 0.6000 Accuracy: 0.8033 Time: 8.77304  | Val Loss: 0.7567 Accuracy: 0.7847\n",
      "Epoch: 22/130 Train Loss: 0.6120 Accuracy: 0.8030 Time: 8.96509  | Val Loss: 0.5887 Accuracy: 0.8163\n",
      "Epoch: 23/130 Train Loss: 0.5908 Accuracy: 0.8120 Time: 8.81922  | Val Loss: 0.6184 Accuracy: 0.8046\n",
      "Epoch: 24/130 Train Loss: 0.5973 Accuracy: 0.8082 Time: 8.92403  | Val Loss: 0.6259 Accuracy: 0.8066\n",
      "Epoch: 25/130 Train Loss: 0.5932 Accuracy: 0.8083 Time: 8.75295  | Val Loss: 0.5431 Accuracy: 0.8316\n",
      "Epoch: 26/130 Train Loss: 0.5792 Accuracy: 0.8075 Time: 8.73494  | Val Loss: 0.5813 Accuracy: 0.8153\n",
      "Epoch: 27/130 Train Loss: 0.5577 Accuracy: 0.8147 Time: 8.92122  | Val Loss: 0.7744 Accuracy: 0.7605\n",
      "Epoch: 28/130 Train Loss: 0.5645 Accuracy: 0.8214 Time: 8.77162  | Val Loss: 0.5317 Accuracy: 0.8298\n",
      "Epoch: 29/130 Train Loss: 0.5537 Accuracy: 0.8224 Time: 8.48564  | Val Loss: 0.4934 Accuracy: 0.8431\n",
      "Epoch: 30/130 Train Loss: 0.5296 Accuracy: 0.8295 Time: 8.48023  | Val Loss: 0.6071 Accuracy: 0.8110\n",
      "Epoch: 31/130 Train Loss: 0.4207 Accuracy: 0.8652 Time: 8.54079  | Val Loss: 0.3544 Accuracy: 0.8889\n",
      "Epoch: 32/130 Train Loss: 0.3557 Accuracy: 0.8892 Time: 8.74071  | Val Loss: 0.3494 Accuracy: 0.8915\n",
      "Epoch: 33/130 Train Loss: 0.3315 Accuracy: 0.8929 Time: 8.50916  | Val Loss: 0.3464 Accuracy: 0.8904\n",
      "Epoch: 34/130 Train Loss: 0.3240 Accuracy: 0.8949 Time: 8.49545  | Val Loss: 0.3485 Accuracy: 0.8927\n",
      "Epoch: 35/130 Train Loss: 0.3094 Accuracy: 0.8980 Time: 8.54911  | Val Loss: 0.3260 Accuracy: 0.8986\n",
      "Epoch: 36/130 Train Loss: 0.2974 Accuracy: 0.9061 Time: 8.67080  | Val Loss: 0.3394 Accuracy: 0.8938\n",
      "Epoch: 37/130 Train Loss: 0.2920 Accuracy: 0.9064 Time: 8.49001  | Val Loss: 0.3358 Accuracy: 0.8950\n",
      "Epoch: 38/130 Train Loss: 0.2829 Accuracy: 0.9102 Time: 8.78764  | Val Loss: 0.3305 Accuracy: 0.9004\n",
      "Epoch: 39/130 Train Loss: 0.2840 Accuracy: 0.9072 Time: 8.76745  | Val Loss: 0.3313 Accuracy: 0.8971\n",
      "Epoch: 40/130 Train Loss: 0.2684 Accuracy: 0.9122 Time: 8.87238  | Val Loss: 0.3309 Accuracy: 0.8983\n",
      "Epoch: 41/130 Train Loss: 0.2769 Accuracy: 0.9127 Time: 8.87495  | Val Loss: 0.3320 Accuracy: 0.8981\n",
      "Epoch: 42/130 Train Loss: 0.2792 Accuracy: 0.9091 Time: 8.83704  | Val Loss: 0.3245 Accuracy: 0.9017\n",
      "Epoch: 43/130 Train Loss: 0.2585 Accuracy: 0.9152 Time: 8.94195  | Val Loss: 0.3236 Accuracy: 0.9006\n",
      "Epoch: 44/130 Train Loss: 0.2455 Accuracy: 0.9221 Time: 8.74977  | Val Loss: 0.3257 Accuracy: 0.9011\n",
      "Epoch: 45/130 Train Loss: 0.2654 Accuracy: 0.9122 Time: 8.81821  | Val Loss: 0.3322 Accuracy: 0.8968\n",
      "Epoch: 46/130 Train Loss: 0.2624 Accuracy: 0.9161 Time: 8.69036  | Val Loss: 0.3232 Accuracy: 0.9045\n",
      "Epoch: 47/130 Train Loss: 0.2373 Accuracy: 0.9224 Time: 8.75307  | Val Loss: 0.3300 Accuracy: 0.9001\n",
      "Epoch: 48/130 Train Loss: 0.2414 Accuracy: 0.9222 Time: 9.17338  | Val Loss: 0.3300 Accuracy: 0.9019\n",
      "Epoch: 49/130 Train Loss: 0.2514 Accuracy: 0.9197 Time: 8.78280  | Val Loss: 0.3421 Accuracy: 0.8976\n",
      "Epoch: 50/130 Train Loss: 0.2360 Accuracy: 0.9254 Time: 8.58973  | Val Loss: 0.3397 Accuracy: 0.9004\n",
      "Epoch: 51/130 Train Loss: 0.2164 Accuracy: 0.9311 Time: 8.66292  | Val Loss: 0.3375 Accuracy: 0.8994\n",
      "Epoch: 52/130 Train Loss: 0.2315 Accuracy: 0.9230 Time: 8.66577  | Val Loss: 0.3561 Accuracy: 0.8963\n",
      "Epoch: 53/130 Train Loss: 0.2159 Accuracy: 0.9292 Time: 9.26833  | Val Loss: 0.3320 Accuracy: 0.9050\n",
      "Epoch: 54/130 Train Loss: 0.2316 Accuracy: 0.9257 Time: 8.57988  | Val Loss: 0.3482 Accuracy: 0.8983\n",
      "Epoch: 55/130 Train Loss: 0.2179 Accuracy: 0.9328 Time: 8.83751  | Val Loss: 0.3537 Accuracy: 0.9034\n",
      "Epoch: 56/130 Train Loss: 0.2282 Accuracy: 0.9249 Time: 8.80560  | Val Loss: 0.3317 Accuracy: 0.9006\n",
      "Epoch: 57/130 Train Loss: 0.2097 Accuracy: 0.9308 Time: 8.69217  | Val Loss: 0.3365 Accuracy: 0.9024\n",
      "Epoch: 58/130 Train Loss: 0.2103 Accuracy: 0.9340 Time: 8.64119  | Val Loss: 0.3405 Accuracy: 0.8981\n",
      "Epoch: 59/130 Train Loss: 0.2180 Accuracy: 0.9291 Time: 8.74241  | Val Loss: 0.3413 Accuracy: 0.9019\n",
      "Epoch: 60/130 Train Loss: 0.2064 Accuracy: 0.9340 Time: 8.74117  | Val Loss: 0.3556 Accuracy: 0.8953\n",
      "Epoch: 61/130 Train Loss: 0.1804 Accuracy: 0.9414 Time: 8.86285  | Val Loss: 0.3357 Accuracy: 0.9037\n",
      "Epoch: 62/130 Train Loss: 0.1912 Accuracy: 0.9393 Time: 8.81350  | Val Loss: 0.3303 Accuracy: 0.9047\n",
      "Epoch: 63/130 Train Loss: 0.1816 Accuracy: 0.9439 Time: 8.81953  | Val Loss: 0.3318 Accuracy: 0.9039\n",
      "Epoch: 64/130 Train Loss: 0.1778 Accuracy: 0.9428 Time: 8.90395  | Val Loss: 0.3275 Accuracy: 0.9057\n",
      "Epoch: 65/130 Train Loss: 0.1812 Accuracy: 0.9413 Time: 8.95367  | Val Loss: 0.3219 Accuracy: 0.9052\n",
      "Epoch: 66/130 Train Loss: 0.1775 Accuracy: 0.9465 Time: 9.06537  | Val Loss: 0.3236 Accuracy: 0.9085\n",
      "Epoch: 67/130 Train Loss: 0.1827 Accuracy: 0.9420 Time: 8.68214  | Val Loss: 0.3258 Accuracy: 0.9083\n",
      "Epoch: 68/130 Train Loss: 0.1728 Accuracy: 0.9469 Time: 8.75859  | Val Loss: 0.3211 Accuracy: 0.9080\n",
      "Epoch: 69/130 Train Loss: 0.1736 Accuracy: 0.9450 Time: 9.16793  | Val Loss: 0.3207 Accuracy: 0.9060\n",
      "Epoch: 70/130 Train Loss: 0.1650 Accuracy: 0.9469 Time: 8.97208  | Val Loss: 0.3118 Accuracy: 0.9096\n",
      "Epoch: 71/130 Train Loss: 0.1603 Accuracy: 0.9486 Time: 8.65000  | Val Loss: 0.3185 Accuracy: 0.9065\n",
      "Epoch: 72/130 Train Loss: 0.1674 Accuracy: 0.9434 Time: 8.85025  | Val Loss: 0.3155 Accuracy: 0.9075\n",
      "Epoch: 73/130 Train Loss: 0.1604 Accuracy: 0.9487 Time: 8.60692  | Val Loss: 0.3164 Accuracy: 0.9065\n",
      "Epoch: 74/130 Train Loss: 0.1679 Accuracy: 0.9447 Time: 8.71588  | Val Loss: 0.3169 Accuracy: 0.9068\n",
      "Epoch: 75/130 Train Loss: 0.1613 Accuracy: 0.9487 Time: 8.63872  | Val Loss: 0.3142 Accuracy: 0.9083\n",
      "Epoch: 76/130 Train Loss: 0.1651 Accuracy: 0.9475 Time: 8.72542  | Val Loss: 0.3217 Accuracy: 0.9034\n",
      "Epoch: 77/130 Train Loss: 0.1606 Accuracy: 0.9514 Time: 8.56735  | Val Loss: 0.3256 Accuracy: 0.9042\n",
      "Epoch: 78/130 Train Loss: 0.1697 Accuracy: 0.9436 Time: 8.68249  | Val Loss: 0.3307 Accuracy: 0.9042\n",
      "Epoch: 79/130 Train Loss: 0.1568 Accuracy: 0.9503 Time: 8.68518  | Val Loss: 0.3242 Accuracy: 0.9019\n",
      "Epoch: 80/130 Train Loss: 0.1577 Accuracy: 0.9504 Time: 8.82455  | Val Loss: 0.3224 Accuracy: 0.9068\n",
      "Epoch: 81/130 Train Loss: 0.1692 Accuracy: 0.9467 Time: 9.20750  | Val Loss: 0.3134 Accuracy: 0.9085\n",
      "Epoch: 82/130 Train Loss: 0.1544 Accuracy: 0.9511 Time: 9.00948  | Val Loss: 0.3142 Accuracy: 0.9062\n",
      "Epoch: 83/130 Train Loss: 0.1665 Accuracy: 0.9469 Time: 9.05371  | Val Loss: 0.3149 Accuracy: 0.9088\n",
      "Epoch: 84/130 Train Loss: 0.1671 Accuracy: 0.9459 Time: 9.10462  | Val Loss: 0.3200 Accuracy: 0.9068\n",
      "Epoch: 85/130 Train Loss: 0.1672 Accuracy: 0.9490 Time: 8.76361  | Val Loss: 0.3240 Accuracy: 0.9075\n",
      "Epoch: 86/130 Train Loss: 0.1704 Accuracy: 0.9447 Time: 8.84876  | Val Loss: 0.3188 Accuracy: 0.9062\n",
      "Epoch: 87/130 Train Loss: 0.1634 Accuracy: 0.9466 Time: 8.72354  | Val Loss: 0.3179 Accuracy: 0.9073\n",
      "Epoch: 88/130 Train Loss: 0.1674 Accuracy: 0.9451 Time: 8.92157  | Val Loss: 0.3279 Accuracy: 0.9073\n",
      "Epoch: 89/130 Train Loss: 0.1626 Accuracy: 0.9517 Time: 8.72655  | Val Loss: 0.3174 Accuracy: 0.9090\n",
      "Epoch: 90/130 Train Loss: 0.1573 Accuracy: 0.9495 Time: 8.76621  | Val Loss: 0.3153 Accuracy: 0.9080\n",
      "Epoch: 91/130 Train Loss: 0.1544 Accuracy: 0.9526 Time: 8.76063  | Val Loss: 0.3138 Accuracy: 0.9103\n",
      "Epoch: 92/130 Train Loss: 0.1514 Accuracy: 0.9527 Time: 9.08828  | Val Loss: 0.3190 Accuracy: 0.9047\n",
      "Epoch: 93/130 Train Loss: 0.1597 Accuracy: 0.9500 Time: 8.72000  | Val Loss: 0.3241 Accuracy: 0.9093\n",
      "Epoch: 94/130 Train Loss: 0.1563 Accuracy: 0.9499 Time: 8.68459  | Val Loss: 0.3199 Accuracy: 0.9062\n",
      "Epoch: 95/130 Train Loss: 0.1586 Accuracy: 0.9486 Time: 8.78748  | Val Loss: 0.3223 Accuracy: 0.9062\n",
      "Epoch: 96/130 Train Loss: 0.1510 Accuracy: 0.9516 Time: 8.82123  | Val Loss: 0.3228 Accuracy: 0.9060\n",
      "Epoch: 97/130 Train Loss: 0.1463 Accuracy: 0.9541 Time: 8.86497  | Val Loss: 0.3225 Accuracy: 0.9047\n",
      "Epoch: 98/130 Train Loss: 0.1485 Accuracy: 0.9525 Time: 8.90483  | Val Loss: 0.3168 Accuracy: 0.9068\n",
      "Epoch: 99/130 Train Loss: 0.1593 Accuracy: 0.9495 Time: 8.78345  | Val Loss: 0.3196 Accuracy: 0.9096\n",
      "Epoch: 100/130 Train Loss: 0.1583 Accuracy: 0.9513 Time: 8.87840  | Val Loss: 0.3192 Accuracy: 0.9088\n",
      "Epoch: 101/130 Train Loss: 0.1606 Accuracy: 0.9507 Time: 8.72635  | Val Loss: 0.3194 Accuracy: 0.9075\n",
      "Epoch: 102/130 Train Loss: 0.1507 Accuracy: 0.9542 Time: 8.74493  | Val Loss: 0.3156 Accuracy: 0.9080\n",
      "Epoch: 103/130 Train Loss: 0.1527 Accuracy: 0.9522 Time: 8.89855  | Val Loss: 0.3210 Accuracy: 0.9057\n",
      "Epoch: 104/130 Train Loss: 0.1563 Accuracy: 0.9486 Time: 8.95237  | Val Loss: 0.3193 Accuracy: 0.9078\n",
      "Epoch: 105/130 Train Loss: 0.1556 Accuracy: 0.9517 Time: 9.01972  | Val Loss: 0.3179 Accuracy: 0.9101\n",
      "Epoch: 106/130 Train Loss: 0.1537 Accuracy: 0.9515 Time: 8.85401  | Val Loss: 0.3181 Accuracy: 0.9075\n",
      "Epoch: 107/130 Train Loss: 0.1504 Accuracy: 0.9517 Time: 9.12152  | Val Loss: 0.3143 Accuracy: 0.9103\n",
      "Epoch: 108/130 Train Loss: 0.1470 Accuracy: 0.9555 Time: 9.34913  | Val Loss: 0.3192 Accuracy: 0.9062\n",
      "Epoch: 109/130 Train Loss: 0.1512 Accuracy: 0.9514 Time: 8.90960  | Val Loss: 0.3186 Accuracy: 0.9108\n",
      "Epoch: 110/130 Train Loss: 0.1471 Accuracy: 0.9536 Time: 9.08455  | Val Loss: 0.3170 Accuracy: 0.9088\n",
      "Epoch: 111/130 Train Loss: 0.1524 Accuracy: 0.9497 Time: 9.00922  | Val Loss: 0.3165 Accuracy: 0.9108\n",
      "Epoch: 112/130 Train Loss: 0.1523 Accuracy: 0.9519 Time: 9.03412  | Val Loss: 0.3184 Accuracy: 0.9068\n",
      "Epoch: 113/130 Train Loss: 0.1586 Accuracy: 0.9496 Time: 8.93155  | Val Loss: 0.3183 Accuracy: 0.9070\n",
      "Epoch: 114/130 Train Loss: 0.1584 Accuracy: 0.9489 Time: 8.87994  | Val Loss: 0.3198 Accuracy: 0.9070\n",
      "Epoch: 115/130 Train Loss: 0.1463 Accuracy: 0.9542 Time: 9.16409  | Val Loss: 0.3165 Accuracy: 0.9093\n",
      "Epoch: 116/130 Train Loss: 0.1419 Accuracy: 0.9561 Time: 9.07754  | Val Loss: 0.3148 Accuracy: 0.9080\n",
      "Epoch: 117/130 Train Loss: 0.1563 Accuracy: 0.9509 Time: 8.92497  | Val Loss: 0.3168 Accuracy: 0.9078\n",
      "Epoch: 118/130 Train Loss: 0.1523 Accuracy: 0.9524 Time: 8.79228  | Val Loss: 0.3163 Accuracy: 0.9065\n",
      "Epoch: 119/130 Train Loss: 0.1476 Accuracy: 0.9550 Time: 8.81235  | Val Loss: 0.3123 Accuracy: 0.9093\n",
      "Epoch: 120/130 Train Loss: 0.1581 Accuracy: 0.9493 Time: 8.94728  | Val Loss: 0.3213 Accuracy: 0.9083\n",
      "Epoch: 121/130 Train Loss: 0.1554 Accuracy: 0.9500 Time: 8.77298  | Val Loss: 0.3218 Accuracy: 0.9073\n",
      "Epoch: 122/130 Train Loss: 0.1460 Accuracy: 0.9521 Time: 8.95833  | Val Loss: 0.3184 Accuracy: 0.9078\n",
      "Epoch: 123/130 Train Loss: 0.1547 Accuracy: 0.9502 Time: 8.96260  | Val Loss: 0.3192 Accuracy: 0.9083\n",
      "Epoch: 124/130 Train Loss: 0.1518 Accuracy: 0.9522 Time: 8.87919  | Val Loss: 0.3232 Accuracy: 0.9083\n",
      "Epoch: 125/130 Train Loss: 0.1485 Accuracy: 0.9518 Time: 8.71271  | Val Loss: 0.3166 Accuracy: 0.9073\n",
      "Epoch: 126/130 Train Loss: 0.1489 Accuracy: 0.9531 Time: 9.03194  | Val Loss: 0.3188 Accuracy: 0.9106\n",
      "Epoch: 127/130 Train Loss: 0.1550 Accuracy: 0.9508 Time: 8.82488  | Val Loss: 0.3259 Accuracy: 0.9052\n",
      "Epoch: 128/130 Train Loss: 0.1584 Accuracy: 0.9513 Time: 8.98555  | Val Loss: 0.3186 Accuracy: 0.9065\n",
      "Epoch: 129/130 Train Loss: 0.1466 Accuracy: 0.9517 Time: 9.01113  | Val Loss: 0.3179 Accuracy: 0.9093\n",
      "Epoch: 130/130 Train Loss: 0.1592 Accuracy: 0.9489 Time: 8.77789  | Val Loss: 0.3280 Accuracy: 0.9062\n",
      "#Parameter: 1062276 Accuracy: 0.9062420382165605\n"
     ]
    }
   ],
   "source": [
    "from hdd.models.cnn.densenet import (\n",
    "    DenseNetBC121,\n",
    ")\n",
    "\n",
    "net = DenseNetBC121(num_classes=10, dropout=0.0, growth_rate=12).to(DEVICE)\n",
    "dense40_stats = train_net(\n",
    "    net,\n",
    "    train_dataloader,\n",
    "    val_dataloader,\n",
    "    lr=0.1,\n",
    "    weight_decay=1e-4,\n",
    ")\n",
    "\n",
    "eval_result = eval_image_classifier(net, val_dataloader.dataset, DEVICE)\n",
    "ss = [result.gt_label == result.predicted_label for result in eval_result]\n",
    "print(f\"#Parameter: {count_trainable_parameter(net)} Accuracy: {sum(ss) / len(ss)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f39431e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pytorch-cu124",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
